Summary

The apps.analysis JSON column previously encoded four different things at once:

• NULL = queued
• {success:false, logs:'Processing in progress', timestamp} = worker lock
• {success:false, reason, retryable} = failure record
• anything else = completed analysis payload

This PR moves queue/lock/failure state into real columns and drives all scheduling off them, while keeping apps.analysis as the untouched raw analyser payload.

State machine

New columns on apps (migration 007):

• status text NOT NULL DEFAULT 'queued' CHECK IN (queued, processing, analysed, failed)
• processing_started timestamptz — when the current processing lock was taken
• failure_reason text, failure_retryable boolean — populated only for failed

Transitions:

• addApp inserts a row → queued (column default).
• nextApp picks a candidate (queued, expired processing lock, stale analysed, or retryable failed) with FOR UPDATE SKIP LOCKED, then sets status='processing', processing_started=NOW(). This lock is non-destructive — it no longer overwrites analysis, so the last good result stays visible on the website during a refetch.
• updateAnalysis writes the new payload and sets status='analysed' or status='failed' (+ failure_reason/failure_retryable) via the pure, unit-tested deriveAnalysisState helper. It still writes the raw payload to apps.analysis and still snapshots history into app_analyses (ON CONFLICT DO NOTHING).

Migrations & safety

Migrations run on deploy before the new code boots, and must tolerate the currently deployed code running briefly against the migrated schema. All three are additive/backward-compatible:

• 007 — additive columns with defaults + a backfill from the existing JSON state (order matters: the processing indicator also carries success=false, so it is classified first). Malformed processing timestamps are regex-validated and fall back to NOW() so a bad value can't abort the migration. Partial indexes back the queue scan. Old code ignores these columns.
• 008 — json→jsonb and timestamp→timestamptz (values assumed UTC via AT TIME ZONE 'UTC'). The pg node driver treats json/jsonb and timestamp/timestamptz equivalently on both read and write, so the still-running old code keeps working.
• 009 — enforce case-insensitive bundle IDs: merge rows differing only by case (dedupe colliding app_analyses rows around the (appid, analysed) unique index before repointing history to the keeper), then a unique index on lower(appid). No appid values are rewritten, so the existing validated apps_appid_valid CHECK still holds.

Migration ordering 007 → 008 → 009 is enforced by the runner's sorted-filename ledger.

During the brief old-code window a row may end up with status='queued' but an old Processing in progress JSON marker (or vice-versa); this self-heals on the next nextApp and is harmless (idempotent, SKIP LOCKED).

API freeze — Raspberry Pi analyser requires NO changes

The analyser talks to this server only over HTTP. All endpoints it uses are byte-identical in shape:

• /queue still returns app.appid (nextApp returns {appid}).
• /uploadAnalysis and /reportAnalysisFailure still flow through updateAnalysis, which returns the same pg result (RETURNING appid, details, analysed) and still writes the failure JSON to apps.analysis so the website's failure display keeps working.
• /ping unchanged.

Scripts

queue-refetch, priority-report, and queue-status now classify state via the status columns. queue-refetch also resets status='queued' and clears the lock/failure columns when queueing a refetch.

Tests

npm test (node --test): 22/22 pass, including new deriveAnalysisState coverage. No local Postgres was available, so the migration SQL was validated by careful review rather than execution.

🤖 Generated with Claude Code